Starts work on adding recipe upgrade feature #539

schuylermartin45 · 2024-05-16T20:34:31Z

Description

First-pass implementation supporting the V1 recipe standard (used by rattler-build, as discussed in #486

~~As of writing this PR (currently in the DRAFT stage), there are a few issues:~~
1. conda-recipe-manager is not currently in the conda-forge channel. There is an open PR waiting for a review to add it in.
2. Locally, I can't get ruamel to dump souschef's version of the recipe data into a StringIO stream. I had many issues getting conda to use my local copy of conda-recipe-manager and I think it's related to that. I have ruamel.yaml.jinja installed in the local environment, but I still get an error. Here are some logs for more details:

(grayskull) smartin@Schuylers-MacBook-Pro:grayskull> grayskull pypi -u types-toml                       



#### Initializing recipe for types-toml (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package types-toml
types-toml 100% Time:  0:00:00   1.8 MiB/s|####################################################################################|
Checking for pyproject.toml
pyproject.toml not found.
Recovering information from setup.py
Executing injected distutils...
Recovering metadata from setup.cfg
No data was recovered from setup.py. Forcing to execute the setup.py as script
Recovering metadata from setup.cfg
Checking >> -- 100% |#                                                                                  |[Elapsed Time: 0:00:00]
Recovering license info from spdx.org ...
INFO:Best match for license Apache-2.0 was Apache-2.0.
Best matches: [('Apache-2.0', 100.0, 0), ('Apache-1.0', 90.0, 1), ('Apache-1.1', 80.0, 2), ('Apache License 1.0', 67.5, 4), ('Zend-2.0', 55.55555555555556, 3)]
INFO:Github url: https://api.github.com/repos/python/typeshed/license?ref=0.10.8.20240310 - recovering license info
Recovering license information from github...
Recovering license info from repository...
Cloning into '/var/folders/mj/y6v4_y555yx3_370k4sk0_bh0000gp/T/gs-clone-repo-amc55x71'...
fatal: Remote branch 0.10.8.20240310 not found in upstream origin
Recovering license info from repository...
Cloning into '/var/folders/mj/y6v4_y555yx3_370k4sk0_bh0000gp/T/gs-clone-repo-a_k02yuu'...
fatal: Remote branch v0.10.8.20240310 not found in upstream origin
License type: Apache-2.0
License file: []
Build requirements:
  <none>
Host requirements:
  - python >=3.8
  - pip
Run requirements:
  - python >=3.8

RED: Package names not available on conda-forge
YELLOW: PEP-725 PURLs that did not map to known package
GREEN: Packages available on conda-forge

Maintainers:
   - schuylermartin45
Traceback (most recent call last):
  File "/Users/smartin/miniconda3/envs/grayskull/bin/grayskull", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/smartin/work/grayskull/grayskull/main.py", line 295, in main
    generate_recipes_from_list(args.pypi_packages, args)
  File "/Users/smartin/work/grayskull/grayskull/main.py", line 340, in generate_recipes_from_list
    generate_recipe(recipe, config, args.output, args.use_v1_format)
  File "/Users/smartin/work/grayskull/grayskull/utils.py", line 230, in generate_recipe
    upgrade_v0_recipe_to_v1(recipe, recipe_path)
  File "/Users/smartin/work/grayskull/grayskull/utils.py", line 247, in upgrade_v0_recipe_to_v1
    yaml.dump(recipe.yaml, recipe_stream)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 574, in dump
    return self.dump_all([data], stream, transform=transform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 583, in dump_all
    self._context_manager.dump(data)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 913, in dump
    self.init_output(data)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 906, in init_output
    self._yaml.get_serializer_representer_emitter(self._output, tlca)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 644, in get_serializer_representer_emitter
    self.emitter.stream = stream
    ^^^^^^^^^^^^^^^^^^^
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/jinja2/__plug_in__.py", line 88, in stream
    self, Rewriter(val, getattr(self.dumper, '_plug_in_' + typ))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'

The package that this commit relies on is not currently available in conda-forge. Viewer discretion is advised. - First-pass attempt at integrating `conda-recipe-manager` into Grayskull for the purposes of generating recipes in the V1 format.

schuylermartin45 · 2024-06-03T14:43:41Z

Update: I rebuilt my local environment this morning now that conda-recipe-manager is in conda-forge.

Unfortunately I am still getting:

AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'

when I try to run the conversion flag. Does anyone know if I have a missing dependency locally? Do I have to install a plugin not listed in the environment.yaml file for ruamel?

schuylermartin45 · 2024-06-03T14:48:45Z

Update: I rebuilt my local environment this morning now that conda-recipe-manager is in conda-forge.

Unfortunately I am still getting:
AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'
when I try to run the conversion flag. Does anyone know if I have a missing dependency locally? Do I have to install a plugin not listed in the environment.yaml file for ruamel?

Here are the ruamel packages I have installed in this environment:

ruamel.yaml               0.18.6          py312he37b823_0    conda-forge
ruamel.yaml.clib          0.2.8           py312he37b823_0    conda-forge
ruamel.yaml.jinja2        0.2.4                      py_1    conda-forge

schuylermartin45 · 2024-06-03T16:26:03Z

@marcelotrevisani I'd hate to bother you directly, but have you seen these issues before? I get the feeling that I'm missing some dependencies for ruamel and haven't found much when searching with this error message.

Here's how I've configured my development:

conda env create -f environment.yaml --name grayskull --yes
conda run --name grayskull pip install -e .
conda activate grayskull

I've also tried forcing the environment to use 3.11 by editing environment.yaml:

  - python >=3.8,<3.12

marcelotrevisani · 2024-06-03T16:58:24Z

Hi, sorry, I have been busy in my personal life and job.
I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

schuylermartin45 · 2024-06-03T17:53:56Z

Hi, sorry, I have been busy in my personal life and job. I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

No I have not and did not think to do that. Do you know off the top of your head what the last known good version was? I don't see it listed in the environment file so I'm guessing it transitively is included with sous-chef?

schuylermartin45 · 2024-06-03T17:54:49Z

Hi, sorry, I have been busy in my personal life and job. I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

No I have not and did not think to do that. Do you know off the top of your head what the last known good version was? I don't see it listed in the environment file so I'm guessing it transitively is included with sous-chef?

Yeah I see it in sous-chef as >=0.15.3 so I'll start from there and see what happens. Thanks!

schuylermartin45 · 2024-06-05T14:33:49Z

Latest update: I've struggled to get conda to use an older version of ruamel.yaml so I have started to use venv and pip to experiment. The oldest version of ruamel.yaml I've used so far is 0.16.10 in this setup and I still get the same error message. pip would not allow me to install 0.15.3 so I'm going to try to see what the oldest version between the two is that pip will let me revert to.

I also ran a brief experiment yesterday and tried to have ruamel dump to a file and not a string stream. That causes the same error, but locally souschef's library is performing the same logic on my local environment without issue. I get the impression that somehow, when called from grayskull, ruamel can't find the jinja2 plugin that is readily accessible in souschef.

schuylermartin45 · 2024-06-05T14:41:50Z

pip will not let me install ruamel@0.15.* but will let me try ruamel@0.16.* and 0.16.1 returns the same error. I think I've ruled out a version incompatibility issue at this point.

dhirschfeld · 2024-06-05T20:22:42Z

IIRC if you conda install a local .bz2/.conda file it will just do it, ignoring all constraints.

schuylermartin45 · 2024-06-18T19:21:54Z

After talking with @beeankha, we found a workaround. I'm not super proud of it, but I've gotten around the JINJA issue by reading the recipe file produced by souschef::recipe.save(), converting that string, and then rewriting the file with the converted text.

Like I said, it's a hack but it works for the time being. Arguably we could modify this further by having the upgrade option return both a meta.yaml and a recipe.yaml file as we transition between formats.

I'll be moving the PR out of the Draft state now that we have a working solution.

schuylermartin45 mentioned this pull request May 16, 2024

[FEATURE] Add support for new output format #486

Open

Forgot to uncomment line in environment file before making draft PR

94c5ab5

Adds missing preprocessor phase

e6f67fd

Fixes ruamel issue with a workaround

629a925

schuylermartin45 marked this pull request as ready for review June 18, 2024 19:22

schuylermartin45 requested a review from a team as a code owner June 18, 2024 19:22

marcelotrevisani approved these changes Jul 28, 2024

View reviewed changes

marcelotrevisani merged commit 3b83710 into conda:main Jul 28, 2024
5 of 9 checks passed

schuylermartin45 deleted the smartin_adding_crm_conversion_support branch July 29, 2024 15:48

beenje mentioned this pull request Aug 16, 2024

Fix meta.yaml output directory #550

Closed

branchvincent mentioned this pull request Aug 21, 2024

[BUG] PyPI package missing dependency on conda_recipe_manager #554

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Starts work on adding recipe upgrade feature #539

Starts work on adding recipe upgrade feature #539

schuylermartin45 commented May 16, 2024 •

edited

Loading

schuylermartin45 commented Jun 3, 2024

schuylermartin45 commented Jun 3, 2024

schuylermartin45 commented Jun 3, 2024

marcelotrevisani commented Jun 3, 2024

schuylermartin45 commented Jun 3, 2024

schuylermartin45 commented Jun 3, 2024

schuylermartin45 commented Jun 5, 2024 •

edited

Loading

schuylermartin45 commented Jun 5, 2024

dhirschfeld commented Jun 5, 2024 •

edited

Loading

schuylermartin45 commented Jun 18, 2024

Starts work on adding recipe upgrade feature #539

Starts work on adding recipe upgrade feature #539

Conversation

schuylermartin45 commented May 16, 2024 • edited Loading

Description

schuylermartin45 commented Jun 3, 2024

schuylermartin45 commented Jun 3, 2024

schuylermartin45 commented Jun 3, 2024

marcelotrevisani commented Jun 3, 2024

schuylermartin45 commented Jun 3, 2024

schuylermartin45 commented Jun 3, 2024

schuylermartin45 commented Jun 5, 2024 • edited Loading

schuylermartin45 commented Jun 5, 2024

dhirschfeld commented Jun 5, 2024 • edited Loading

schuylermartin45 commented Jun 18, 2024

schuylermartin45 commented May 16, 2024 •

edited

Loading

schuylermartin45 commented Jun 5, 2024 •

edited

Loading

dhirschfeld commented Jun 5, 2024 •

edited

Loading